Center for 
Gece: for, 
Designing a Coherent State System of Accountability: 
The Every Student Succeeds Act and Perkins V 


Erika L. Landl, PhD 
National Center for the Improvement of Educational Assessment 
September 20, 2018 
Introduction 
Over the last five years, the U.S .Department of Education (USED) is requiring states to provide 
more and better evidence supporting the quality and validity of their assessment and accountability 
systems. Notably, states are asked to explain more fully how the proposed design of their systems 
will provide accurate and reliable information that improves student outcomes. One sees this 
expectation in the USED guidance regarding the assembly of materials for assessment peer review 
(U.S. Department of Education, 2015) and the development of state plans for the Every Student 
Succeeds Act (U.S. Department of Education, 2017). For example, the initial ESSA template 
challenged each State Education Agency (SEA) to “reflect on its overall vision and how the different 
sections of the consolidated state plan work together to create one comprehensive approach to 
improving outcomes for all students,” and, further, to articulate how the plan would “serve to meet 


the SEA’s vision with regard to its education system.” 


While this call for coherence traditionally has focused on a state’s response to specific legislation, 
USED recently pushed for improved coherence across state plans. This is evident in ESSA and the 
Strengthening Career and Technical Education for the 21st Century (Perkins V) Act (2018) both of 
which necessitate and support improved alignment with each other and the Workforce Innovation 


and Opportunity Act (WIOA) as demonstrated by the: 


e inclusion of common terms and definitions; 

e required alignment of academic and Career Technical Education (CTE) content standards; 

e increased focus on incorporating, and measuring, the state’s academic content standards in 
CTE courses and programs of study; 

e increased consistency in the indicators and procedures required for accountability; 

e aligned requirements for federal reporting (Advance CTE, n.d.); and 


e increased flexibility afforded to states around the use of federal funds. 


Landl, Erika &e) ® Page 1 


*sCenter for 
Cecenter for 


In its appeal for alignment, USED is acknowledging that these laws, despite their different 
requirements, timelines, and resources, have common goals and objectives that will not be met if a 
state responds to each in isolation (Cushing, E., Therriault, S. and English, D, 2017). Most notably, 
all three laws support the development and maintenance of programs to ensure that students and 
workers acquire the necessary knowledge, skills, and abilities to engage successfully in college and 


careers, ultimately strengthening the workforce. 


Federal efforts to improve alignment are a good start, to be sure. But they are not enough to help 
those who are charged with designing and implementing these programs establish accountability 
provisions that work in a coordinated manner to effectively and efficiently meet a state’s goals. 
This requires not only an understanding of what it means for a system to be coherent, but also a 
reconceptualization of ESSA, Perkins V, and WIOA as complementary elements of a larger state 
system of accountability. Further, since these programs typically are developed, monitored, and 
administered by different offices, SEAs need a strategy that serves to align these efforts. To that 
end, this brief discusses the characteristics and features of a coherent system, outlining ten 
recommendations to support those charged with developing, evaluating, or modifying state plans 


under ESSA, Perkins V, and WIOA. 


What is a coherent system? 
At the most basic level, a coherent system can be described as one in which (a) each element of the 


system is designed to interact with other elements in a specified manner to achieve common goals, 
and (b) there is a clear theory of action undergirding the intended interaction of system elements 


toward this end. 


Thus, a coherent system is 
e purposeful: developed in service of a particular goal; 
e rational: supported by a clear and evidence-based rationale; 
e efficient: each system element has a clear, non-duplicative role complementing that of other 
elements in the system; 
e transparent: primary elements of the system, and their intended role, are concrete and 


intuitive; and 
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e communal: developed through a collaborative process, ensuring that all participants 
understand the goals of the system and have a common vision of how these goals will be 


achieved. 


The role and significance of coherence in system design is addressed frequently in the field of 
educational assessment. For example, the authors of the seminal work Knowing What Students 
Know: The Science and Design of Educational Assessment (NRC, 2001) argue that coherence across 
key components of an educational system (curriculum, assessment, and instruction), undergirded 
by aclear theory of how students acquire knowledge, is required in order for assessments to 
support student learning. Chattergoon and Marion and (2016) extend this argument, citing 
coherence as one of three core criteria states should consider when developing or evaluating a 
balanced assessment system. Specifically, in order for assessment results to support valid 
inferences about, say, student achievement, teacher instruction, or school quality, assessments at 
all levels of the educational system—classroom, school, district, state—must be designed in 
consideration of acommon model of student learning, mirroring that reflected in curriculum and 
instruction. Without a clear model to tie system elements together, assessment results may 
provide conflicting information to students, teachers, parents, and administrators and therefore 


constrain, rather than support, good decision making. 


Similarly, a compilation of independently developed state plans will not result in an accountability 
system that incentivizes collaboration and provides a state and the institutions it serves with 
accurate, consistent information about progress toward desired student outcomes. To have the 
intended impact, SEAs must thoughtfully design their implementations of ESSA, Perkins V, and 
WIOA for coherence by 


e establishing an overarching goal and theory of action for student learning that serves as a 
shared foundation for accountability system design and evaluation (providing for a system 


that is purposeful and rational); 


e identifying common elements across the three laws that are used to support coherence 


(providing for a system that is efficient); and 
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e engaging in strategic, ongoing system design and evaluation activities that include 
representatives from K-12, CTE, workforce, and higher education (providing for a system 


that is transparent and communal). 


Designing for coherence is beneficial on multiple fronts. Not only does it result in systems having 
the desired characteristics, the design process itself necessitates coordination and communication 
between key stakeholders and fosters a shared sense of responsibility for meeting the state’s goals. 
The remainder of this report is structured around ten actionable recommendations: concrete steps 
states can take to promote coherence across a state’s plans for accountability under ESSA, Perkins 
V,and WIOA. While the text and exemplars focus primarily on ESSA and Perkins, due to their 


shared focus on accountability for educational institutions, the overarching recommendations can 


easily be extended to include WIOA. 


As shown Table 1, these recommendations are structured in terms of three factors that influence 


coherence: clarity, consistency, and collaboration. 


Table 1. Recommendations for Establishing Coherence across a State’s Plans for Accountability 


Coherence Factors 


Recommendations 


Clarity - around the 
state's goals, priorities, 
and vision for student 


learning 


1. Articulate the State’s Goals and Theory of Action for Student 


Learning 


Identify and Define Student Outcomes that Represent Attainment 


of the State’s Goal for Student Learning 


Define the Role of Accountability in Supporting the State’s Goal for 


Student Learning 


Consistency - across 
common structural 
elements in service to 
the shared goal(s) and 
vision for student 


learning 


Define and report common indicators in a consistent manner 


Use consistent procedures to establish annual targets for 


performance 


Use common business rules and psychometric approaches to 


address similar design decisions 


Consistently highlight state values and priorities in design 


decisions 


Represent a consistent philosophy related to the role of 


identification and the state’s system of support 
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Collaboration - among 9. Coordinate across state offices to establish a state-wide glossary 
state and local leaders in summarizing terminology specific to the state’s system of 
K-12, CTE, and assessment and accountability 


workforce to ensure that | 10. Bring together state leaders in K-12, CTE and Workforce when 
each state plan is evaluating, developing or revising states plans under ESSA, Perkins 
designed to support V, and WIOA. 

progress toward the 
shared goal and vision 


for student learning 


Recommendations 1-3: Clarify the State’s Goals, Priorities and Vision for 
Student Learning 
At a high level, the accountability requirements under ESSA and Perkins V share several common 


elements (Figure 1). 


Cia late|{or-\ x0) gcmey mst nlolo) M21 acolmaat-laler=¥A@l0r-) lin Core Indicators of Program Quality 


¢ State-Defined Long Term Goals/Annual Targets for State or Locally Defined Levels of Performance 
Performance for Key Indicators lela Golaew latol (ezine) es 


e Identification of Schools Requiring Support and Identification of Eligible Entities/Recipients 
Improvement Requiring Improvement Plans 
e Annual Reporting by Indicator and Sub-Group INT ALU Viste exeladlatcal VallatelCoclcolar-late Asie] mC] aelt]°) 


e State Provision of Support to School in Need State Provision of Support to Systems in Need 


Figure 1. Shared Accountability Elements under ESSA and Perkins V 


Both require a state agency to articulate indicators and associated measures of quality, establish 
annual performance targets for those indicators, identify institutions in need of improvement and 
support, and annually report overall and disaggregated results. Despite these common elements, 
however, accountability provisions designed to support ESSA and Perkins V will not be coherent in 


the absence of three essential conditions: 


1. The state must have a clear goal for student learning and a vision for how that goal will be 


achieved. 


2. The state must identify and clearly define goal-related student outcomes. 
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3. The state must define the accountability system’s intended role in supporting the state’s 


goal for student learning. 


These three conditions collectively define a theory of action, highlighting the features to be valued 
and prioritized when designing the state’s system of accountability. As illustrated in Figure 2 and 
discussed in the sections that follow, a state’s response to these conditions (blue ovals) will 

influence the design decisions (green ovals) in a consistent manner, providing for both within and 


between-plan coherence. 


State Goal for Student Learning 


State Vision, Values and Priorities 


What conditions, What outcomes What role should the state 
information, supports represent accountability system play 
will bring about Elae-lialeal-lalme)mdal= to support attainment of 
attainment of goal? goal? the goal? 


Accountability Considerations (ESSA or Perkins) 


shah Procedures 
Priorities State/Local 


(e.g. equity; Inputs and Reporting 


transparency) Supports 


used to 
support 
iTol=Vaiuhater-}ate)a) 


Figure 2. Accountability System Design Elements Influenced by a State’s Theory of Action 


The need for a clear, state-defined theory of action recurs throughout this paper. Ifa state cannot 
articulate what it is trying to accomplish and, in turn, provide a roadmap for getting there, efforts to 
design for coherence will be futile. The ten recommendations highlight key elements of a state’s 


overarching theory of action and their role in effecting broader system-level coherence. 


1 Soe Hall, E., Domaleski C., Russel, M., and Pinnsonneault, L .. (2016) A Framework to Support Accountability 
Evaluation for a broader discussion of the elements of a theory of action and how they support the design and 
evaluation of accountability systems. Retrieved on September 18, 2018 from 
https://www.centerforassessment.org/sites/default/files/publications/A%20Framework%20to%20Support%20Acc 
ountability%20Evaluation.pdf 
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Recommendation 1: Articulate the State’s Goal for Student Learning 

To facilitate the development of a coherent system of accountability, a state must operationalize its 
desired goal—defining the goal and specifying how its attainment will be determined—for all 
students receiving state-funded educational services. The goal statement should delineate what the 
state intends to accomplish for its students but also be broad enough to resonate with stakeholders 
affiliated with different components of the educational system (e.g., K-12, CTE, workforce, higher 
education). For example, a state department of education may claim its overarching goal for 
student learning is to ensure all students are “ready for college and careers” or will be “successful in 
post-secondary education, training or employment.” While such statements often can be 
interpreted in various ways, the process of operationalizing the goal facilitates communication 
across offices within a state agency and, further, provides the foundation for building a coherent 


system of accountability. 


In addition to simply stating a goal, state leaders must work together to establish a clear vision for 
how this goal will be achieved. Specifically, they must describe the conditions, activities, 
interactions, and initiatives they believe will bring about change, and determine how those 
elements should be represented across state plans. They must collaborate in answering the 
question, “What do we as a state need to focus on to make progress toward meeting this goal in the 
short and long term?” If college and career readiness (CCR) is the desired goal for all students, for 
example, what is most likely to bring about that objective—personalized learning, a supportive 
school climate, more occasions to apply skills within an authentic work-place setting? A state must 
articulate its beliefs and priorities so they are represented clearly and consistently in a state’s 


implementation of ESSA, Perkins V, and WIOA, as appropriate. 


While more prescriptive than ESSA, Perkins V does allow for the inclusion or reporting of 
additional indicators of performance for career and technical education activities the state believes 
will drive progress and provide for the greatest return of investment of federal funds. Therefore, 
important non-mandated indicators (e.g., school climate, community servce) in the state’s theory of 
action should be incorporated into the state’s plan. Further, Perkins V also allows for the inclusion 
of additional measures of student success in CTE, provided they are “valid and reliable.” It is 
through these additional indicators that a state can highlight shared priorities and strengthen the 


influence of those priorities in supporting the attainment of state goals. 
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Recommendation 2: Identify and Define Student Outcomes that Represent Attainment of the 
State’s Goal for Student Learning 
While an overarching goal and vision is necessary, it is not sufficient for supporting the 
development of a coherent accountability system. The SEA also must identify those outcomes it 
believes represent successful attainment of that goal and, further, describe how they will be defined 
and operationalized. For example, if the goal is for all students to be “college and career ready,” the 
associated student-level outcomes endorsed by the state (e.g., meeting proficiency on the state 
assessment, attaining an industry credential) must be clearly defined and consistently represented 


at all levels of the system, as appropriate. 


While different student outcomes may be reported and prioritized under ESSA and Perkins V, how 
the student-level outcomes are defined, measured, and referenced on reports and in state plans 
should not. For example, what it means to be “college-ready” should not be defined by different 
standards (i.e., cut scores on the state test) under ESSA and Perkins V. Similarly, the various 
credentials deemed appropriately rigorous for demonstrating “career readiness” should be the 
same across the two. Working together, state leaders must ensure that all stakeholders understand 


the outcomes that are valued and prioritized and how they are defined. 


In contrast, we do expect different outcomes and measures of system quality to be considered under 
ESSA and Perkins V. That is, each program should identify different indicators/measures to 
evaluate the extent to which schools (under ESSA) or eligible entities/institutions (under Perkins 
V) provide the intended high-quality services and opportunities for effecting desired student 
outcomes. In its plan for Perkins V, for example, a state may include an indicator of participation in 
work-based learning, believing that such participation increases the likelihood of attaining an 
industry credential or related outcome. While perhaps not included in the state’s plan under ESSA, 
this indicator nonetheless supports system coherence if it aligns with the state’s overarching vision 


and supports the attainment of desired student outcomes. 


Recommendation 3: Define the Role of Accountability in Supporting the State’s Goal of 
Student Learning 


From a federal perspective, accountability systems provide a means to assess the effectiveness of a 
state in achieving progress related to its specified goal(s) for student learning. However, each state 


is responsible for determining how the accountability system facilitates this progress. One state 
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may believe that accountability will foster change by motivating educators to work harder, while 
another believes the sole function of the system is to provide formative feedback for improvement. 
And yet another state may view the system as way to provide stakeholders with fair and unbiased 
information about the school performance and quality, or as a means for determining where state 
support for local reform initiatives are most urgently needed. The intended role of the 
accountability system directly influences key design decisions affecting the selection and weighting 
of indicators, procedures for identifying schools/agencies/institutions for support, and the content 
and format of reports. Therefore, a state’s perspective on the intended impact of the accountability 
system must be clear so that viewpoint can be consistently represented across state plans, as 


appropriate. 


Table 2. How the Role of an Accountability System can Influence Design Decisions 


Example of intended role of How influence accountability system design? 

accountability system 

To motivate schools/entities to - Include indicators reflecting the degree to which a 
establish conditions and school/entity/institution provides students access to 
opportunities that provide for opportunities relating to desired outcomes (e.g., high- 
more equitable outcomes quality teachers, courses providing college credit, work- 


based learning or apprenticeship opportunities, a variety 
of programs of study, certifications) 

- Identify schools/entities/institutions that are performing 
far below expectations regarding these indicators. 

- Highlight schools/entities/institutions showing 
significant improvements from the previous year 


To shine a spotlight on gaps in - Establish an indicator that reflects a measure of 

achievement differential performance by subgroup (e.g., on academic 
assessments, industry credential rate, graduation rate, 
employment rate upon exit) 

- Report or reward progress in reducing gaps in subgroup 
performance for selected indicators 

- Require schools/entities/institutions having the largest 
gaps in performance on key indicators to develop 
targeted improvement plans 


To provide information that - Develop reporting systems allowing for data 

helps systems identify areas of disaggregation and comparisons to similar 

strength and need and strategies schools/entities; connect to external resources that 

for improvement. identify evidence-based strategies for improvement on 
indicators for which the school/entity did not meet 
expectations 
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Recommendations 4-8: Strive for Consistency across Common Structural 
Elements 
If we see ESSA and Perkins as two components of a state accountability system designed to support 
a shared goal and vision, then it makes sense that common features across the two laws are defined 
and operationalized in a consistent manner where appropriate. Table 3 highlights four important 
touchpoints—a common element, design feature, or legislative requirement—across ESSA and 
Perkins V that should be considered when designing, evaluating, or modifying state plans for 
coherence. Recommendations accounting for these touchpoints are detailed in the sections that 


follow. 


Table 3. Key Touchpoints across ESSA and Perkins V 


e Performance Indicators: Both Perkins V and ESSA identify performance indicators that serve 
as the basis for evaluating the performance of the state and its eligible recipients (i.e., those that 
receive state funding). Two indicators, academic achievement and graduation rate, are common 
to ESSA and Perkins V. (Appendix A describes the required indicators under ESSA and Perkins V.) 


e Annual Performance Targets: Both ESSA and Perkins V require states to establish annual 
expectations for performance in specified areas. Under ESSA, states must establish long-term 
goals and measures of interim progress for academic achievement, graduation rate, and, in the 
case of English learners, progress toward English language proficiency. Under Perkins V, the 
state and each eligible entity must establish “levels of performance” for each core indicator 
listed in Appendix A, including academic achievement and graduation rate. (Appendix B provides 
the requirements for the specification of performance targets under ESSA and Perkins V.) 


e Annual Reporting against Targets and Indicators: Under ESSA and Perkins V, states and local 
entities are required to report on the performance of all students and for each subgroup for each 
indicator and against established performance targets. States are required to define the 
minimum n-count to be used for reporting results, and explain its determination. 


e Identification of Systems Requiring Support: Both ESSA and Perkins V require states to 
determine the criteria for identifying schools/eligible entities for support. Under ESSA, schools 
can be identified for Targeted Support and Improvement (TSI) or Comprehensive Support and 
Improvement (CSI), each of which has different consequences and requirements. Under Perkins 
V, the state and each eligible recipient not meeting 90% of the defined level of performance on 
each indicator will be required to develop and implement an improvement plan. In all cases, 
criteria for the provision of technical assistance and/or subsequent action are defined by the 
state. (See Appendix B for factors influencing identification under ESSA and Perkins V.) 


Recommendation 4: Define and report common indicators in a consistent manner. 


As shown in Appendix A, both ESSA and Perkins V require an indicator of academic achievement 


and graduation rate. Unless there is a good reason to do otherwise, these indicators should be 
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defined and operationalized in the same way across the different plans. This is important for at 
least two reasons. First, stakeholders can discuss performance and results in a consistent manner, 
minimizing the likelihood of misinterpretation. Second, this sends a clear signal regarding what the 


state believes should be valued and rewarded for purposes of accountability. Consider a state that 


defined academic achievement in Perkins V and ESSA as follows: 


For secondary institutions, achievement School achievement scores are 

scores are calculated as the percentage of calculated as a weighted sum across 
CTE concentrators meeting or exceeding student scores where each achievement 
proficiency on the state’s grade-level level is associated with a different point 
assessments for math and English language __ value (e.g., Does not Meet=0; Partially 
arts. Meets=.5; Meets=1; Exceeds=1.25,) 


Under this scenario, a CTE center with a 67% achievement score and a high school with 67% 
achievement score may be misinterpreted as having identical performance, when this clearly is not 
the case, as demonstrated in the top half of Table 4. Similarly, a CTE center and a high school 
having the same distribution of student performance on the state assessment will receive different 


achievement scores as shown in the bottom half of this table. 


Table 4. Impact of Different Indicator Definitions on Achievement Scores 


Performance Level 
Does | Partially | Meets | Exceeds 
not Meets 

meet 
Different CTE 13% 20% 40% 27% 
Performance Center A 
Distribution/Same | High 22% 32% 27% 19% 
Score School A 
Same Performance | CTE 10% 40% 40% 10% 
Distribution / Center B 
Different Score High 10% 40% 40% 10% 

School B 
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Using the ESSA calculation, High School B earns a score that is 22.5 points higher than CTE Center B 


even though both met/exceeded the standard at a rate of 50%. 


While computational differences across plans were not a major concern in the past, states are 
diligently working to establish comprehensive accountability dashboards that allow for these types 
of results to be presented side by side. In the case of the example above, without explicit reference 
to the calculations used and how/why they differ, stakeholders would not know that these 
measures cannot be interpreted in a similar manner. Furthermore, in the current example, the 
proposed definitions make different claims regarding what should be valued and awarded when 
evaluating academic achievement. The Perkins definition highlights attainment of proficiency while 
the ESSA definition rewards and incentivizes moving students from one level to the next across the 


full range of the scale. 


Insofar as score misinterpretation can have decidedly undesirable consequences, following 
Recommendation 4 clearly is in the best interests of all. But if a state concludes it is necessary for 
common indicators to be defined differently across state plans, the state should ensure that these 


differences are obvious—and with appropriate caveats—in any reporting of results. 


Recommendation 5: Use consistent procedures to establish annual targets for performance 
As shown in Appendix B, both ESSA and Perkins require states to report performance against 
annual targets in the areas of academic achievement and graduation rate. In the same way these 
indicators should be defined in a consistent manner across state plans, the procedures and 
rationale underlying the specification of annual targets for performance also should be consistent. 
This does not mean the targets need to be the same (although this may be reasonable, depending on 


the state’s theory of action), but nor should they be defined in isolation. 


Under ESSA, for example, states are required to specify long-term goals and measures of interim 
progress for all students and each subgroup. Because secondary CTE concentrators are a subgroup 
of all secondary students, SEAs therefore should consider whether procedures used to determine 
appropriate and feasible targets for subgroups under ESSA can be used to establish annual levels of 
proficiency for CTE concentrators under Perkins V. For example, consider a state that defines 
annual targets for graduation rate based on a long-term goal of 90% for all students and each 
subgroup by 2023. The state acknowledges that student groups starting at lower points in the 


baseline year will need to improve at greater rates to achieve long-term goals, but argues that 
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common goals are necessary to ensure sustained focus on improving the performance of all 
students. If this argument represents the state’s overarching belief and theory of action regarding 
how improved graduation rates will be realized, is it sensible to establish different annual targets 


under Perkins? If the answer is yes, the state should explain its rationale for this decision. 


Recommendation 6: Use common business rules and psychometric approaches to address 
similar design decisions 


There are various technical decisions that must be made to support accountability requirements 
within ESSA and Perkins V. Some of these decisions are essentially business rules (e.g., rounding 
rules, minimum n-counts, aggregation procedures), while other decisions are psychometric (e.g., 
procedures for evaluating alignment, calculating performance gaps, or assessing the reliability and 
validity of indicators). These decisions are not arbitrary or capricious. Rather, they are supported 
by data and/or a compelling argument regarding why one approach is more appropriate than 
another. To the extent possible, states should address common design decisions in the same way 
across ESSA and Perkins V. This not only improves coherence, but insofar as identifying and 
defending a particular approach often requires significant resources, time and money will be saved 


as well. 


Recommendation 7: Consistently highlight state values and priorities 

To send aclear message and carry the greatest impact possible, the priorities and initiatives 
highlighted in a state’s theory of action should be represented across state plans for educational 
accountability. For example, if a state indicates that a supportive climate for student learning is 
necessary to promote student success and therefore puts systems in place to support improvement 
in this regard, a measure of climate should be represented in each state plan. Consistently 
attending to state priorities, and in a way that highlights the state’s vision for improvement, will 
facilitate communication and collaboration among stakeholders and, in turn, increase the likelihood 
of progress toward the state’s goals. In addition, section 113 (b)(2)(B) and (C) of the Strengthening 
Career and Technical Education for the 21st Century Act (2018) states that “in developing core 
indicators of performance, an eligible agency shall, to the greatest extent possible, align the 
indicators so that substantially similar information gathered for other State and Federal programs 
may be used to meet the requirements of this section.” This reflects a desire to have consistency in 
the information provided to stakeholders to inform subsequent decisions about the performance of 


schools/entities/institutions and the state in driving progress toward desired student outcomes. 
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Recommendation 8: Represent a consistent philosophy related to the role of identification 
and the state’s system of support. 


While legislation often dictates how states identify entities that require support or the development 
of an improvement plan, the state is responsible for determining the criteria that ultimately result 
in identification. Under Perkins V, for example, if the state or an eligible recipient of the state fails 
to meet a level of performance that is at least 90% of that expected for each core indicator, the state 
or eligible agency is required to develop and implement a plan for improvement.2 While there are 
requirements underlying the specification of performance levels for core indicators, the state 
defines the annual targets constituting adequate performance.? And under ESSA, the state, in 
consultation with stakeholders, defines the long-term goals and procedures for meaningful 
differentiation resulting in the identification of schools for support (see Appendix B). Different 
design decisions send different messages regarding the role of goals/targets in supporting 
identification. For example, lenient targets and conservative identification criteria throw light ona 
small number of extremely poor performing systems. In contrast, stringent targets and liberal 
identification criteria flag a broader range of at-risk schools. Because identification requires an 
improvement plan and may lead to subsequent action (e.g., more rigorous intervention, 
suppression of funds), how targets are specified reflects a state’s beliefs regarding the utility of 
identification as a means of supporting improvement. If the identification criteria reflect markedly 
different philosophies under ESSA and Perkins V (e.g., flag many vs. flag few), this suggests these 


beliefs vary across programs as well. 


States also are charged with determining the nature of support provided and who is responsible for 
ensuring its effectiveness. Specifically, what are the respective roles of state and local entities in 
supporting local improvement efforts? Will the state engage in review and audit functions, or will it 
focus on developing/distributing resources that allow districts and eligible agencies/institutions to 
implement and evaluate improvement plans at a local level? These decisions impact the resources, 
supports, and infrastructure required at the state and local level and, therefore, must be considered 


when making design decisions. A state that wants to monitor the progress of schools/entities 


* See : Perkins V, Sec 123 - Improvement Plans 

3 Perkins does allow for eligible recipients to either accept the levels of performance established by the state 
or negotiate local levels of performance. Local levels, if proposed must meet a variety of criteria and be 
submitted with data that indicates why they are more appropriate given the characteristics of CTE 
concentrators currently served (See Section 113 (b)(4) (A) 
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identified for support, for example, first will need to establish identification procedures that allow 


for this level of engagement. 


Establishing a common philosophy related to identification and the provision of support that 
extends across ESSA and Perkins V may seem difficult to address in practice. However, if the state 
believes a well-established system of support is the mechanism by which improvement occurs, it 


should reflect a common vision of how this is represented under ESSA and Perkins V. 


Recommendations 9-10: Provide for Collaboration among State Leaders in K- 
12, CTE, and Workforce 


Collaboration is the final coherence factor to be considered. State agencies must collaborate to 
establish systems that work with, not against, one another toward the attainment of a common set 
of state goals. This requires a move away from the traditional, silo-based structure within which 
most education agencies unfortunately exist, to a web-based structure in which connections 
between state offices /divisions and the initiatives they support are developed and nurtured 
through communication, transparency, and a shared understanding of the state’s goals and 
priorities. The final recommendations below represent only two of the many things a state can do 
to facilitate collaboration and improved communication among and between state and local 
agencies. Local education agencies arguably should engage in a similar process to ensure that the 
efforts and initiatives established at a local level support each other and align with the priorities 


and goals defined by the state. 


Recommendation 9: Develop tools and resources that facilitate collaboration and improve 
communication across state agencies 


State departments of education should have tools and resources that facilitate coherence in the 
development of plans, procedures, and documents by different offices and divisions. Consistent 
with federal efforts to align definitions across ESSA, Perkins V, and WIOA, for example, states 
should establish a glossary of state-specific terms, phrases and acronyms. Such a glossary 
promotes not only coherence, but it also improves communication across departments and is 
helpful when responding to legislation. Regarding the interdepartmental communication, for 
example, different levels of performance on the state summative assessment should not be referred 
to as “performance levels” by one division/agency and “proficiency levels” by another. As for 


responding to legislation, college-and-career readiness, for example, should be defined in the same 
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way even if different indicators are reported under ESSA and Perkins V. To support coherence in its 
reports and communications involving results, furthermore, a state should develop shared business 
rules and data systems that facilitate the transfer, combination, and comparison of data across 


divisions/offices within the SEA, where appropriate. 


Recommendation 10: Bring together state leaders in K-12, CTE, and workforce when 
evaluating, developing, or revising states plans for ESSA and Perkins V.4 
To develop and implement a coherent accountability system, state leaders from each education 


agency/division first must have a shared understanding of the state’s current system of 
accountability. This includes the requirements, goals, and decisions underlying the current plans 
for ESSA, Perkins V, and WIOA. Only then can state leaders collectively engage in the critical 
activity of looking across these plans to identify and, in turn, address potential elements of 


incoherence. This activity is part of a larger process that should entail 


e reflecting upon the state’s overarching goal for student learning and broad vision of how it 


will be achieved; 


e discussing the required accountability elements of each law and how they currently are 


addressed in the state’s plans for ESSA, Perkins V, and WIOA; 


e sharing the rationale underlying key design decisions reflected in each state plan, highlighting 


program-specific goals and the priorities these decisions are intended to support; 
e identifying ways the current state plans work with, or against, one another; 
e identifying state-specific touchpoints for improved coherence;> and 


e crafting short- and long-term recommendations for modification and ongoing evaluation, 


consistent with the state’s overarching goals and priorities.® 


Again, design decisions for coherence must be grounded in a shared vision of the state’s goals and 


priorities and a mutual understanding of how to meet those goals. Otherwise, efforts to design for 


° While several touchpoints are addressed in this brief, group discussion likely will identify additional opportunities 
to effect systematic coherence. 

° This bulleted list assumes there is a shared vision for student learning and accountability, as reflected in the top 
half of Figure 2. If not the case, this clearly is the place to start. 
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alignment—fundamental to achieving coherence—will seem arbitrary and senseless, with 


participants in the process reasonably asking “Alignment to what end?” 


Conclusion 


System coherence does not come easy. Rather, it is the product of a thoughtful, intentional design 


process that requires 


e Clarity - around the state's goals, priorities, and theory of action for student learning 


common to K-12, CTE, and Workforce programs; 
e Consistency - in the definition and specification of common elements and procedures, and 


e Collaboration - among state and local leaders in K-12, CTE, and workforce to ensure that 


state plans are designed to be mutually supportive in achieving the state’s goals. 


Over the last several years, SEAs have devoted significant time and effort developing plans for K-12 
accountability that comply with ESSA. This effort, in combination with the reauthorization of 
Perkins, makes this an opportune time for states to review their existing plans and, in turn, 


determine the necessary improvements for establishing a more coherent system of accountability. 
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Appendix A.1: Required Indicators under ESSA, Perkins V, and HR.2353 


ESSA - Secondary Students 


Perkins V 
Secondary CTE Concentrators 


Academic Achievement: 

As measured by proficiency on the annual state 
assessments in Reading, Mathematics, and 
Science. 


Graduation Rate 
4-year adjusted or extended 


English Learner Progress in Achieving 
English Language Proficiency: 

for English learners in each grade 3 through 8 
and the same high school grade in which the 
State assesses for Mathematics and ELA 


School Quality or Student Success: 

At least one measure of school quality or 
student success that is valid, reliable, and 
comparable at a state-level. 


Academic Achievement: Attainment of 
challenging academic standards as measured by 
performance on the state assessments defined in 
ESSA 


Graduation Rate: 
As defined in ESSA 


Post-Secondary Success: 

The percentage of CTE concentrators who, in the 
second quarter after exiting from secondary 
education, are in postsecondary education or 
advanced training, military service or a service 
program that receives assistance under title I of 
the National and Community Service Act of 1990 
(42 U.S.C. 12511 et seq.), are volunteers as 
described in section 5(a) of the Peace Corps Act 
(22 U.S.C. 2504(a)), or are employed. 


CTE Program Quality: 

The percentage of CTE concentrators graduating 
from HS having (at least one of the following to 
be included) 

e attained a recognized post-secondary 
credential; 

e attained post-secondary credits in the 
relevant CTE program or POS earned 
through dual and concurrent enrollment 
program or another transfer agreement; 
or 

e participated in work-based learning. 

May include any other measure of student 
success in career technical education that is 
state-wide, valid and reliable. 


Non-traditional students: 

The percentage of CTE concentrators in CTE 
programs and Program of Study that lead to 
nontraditional fields . 
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Perkins V: Post-Secondary CTE Concentrators 


Post-Program Success: 

Percentage of CTE concentrator who, in the second quarter after program completion, are in 
advanced training, military service, or a service program receiving assistance under Title I of the 
National and Community Service Act of 1990, are volunteers as described in 5(a) of the Peace Corps 
Act, or are placed or retained in employment 


Attainment: 
Percentage of CTE concentrators who receive a recognized post-secondary credential during 
participation within one year of program completion. 


Non-traditional students: 
The percentage of CTE concentrators in CTE programs and or a Program of Study leading to 
nontraditional fields. 
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Appendix B: Annual Performance Targets & Procedures for Identification 


Both ESSA and Perkins V require states to report performance against annual state- 
established targets. Under ESSA, states must establish long term goals and measures of 
interim progress regarding academic achievement, high school graduation rate, and 
increases in the percentage of students making progress in achieving English language 
proficiency (ELP). Under Perkins V, targets must be established for each core indicator 
presented in Appendix A. The first table below summarizes key requirements for the 
specification of annual performance targets under ESSA and Perkins V. The subsequent 
table outlines how schools (under ESSA) and states/eligible recipients (under Perkins V) 


are identified for support and improvement. 


Annual Performance Targets 


ESSA 


Perkins V 


e Long term goals and measures of interim 
progress for academic achievement and 
graduation rate must be established for all 
students and separately for each subgroup.’ 

e Goals for academic achievement must be 
defined in terms of the percentage of 
students achieving proficiency on the annual 
assessments in mathematics and reading or 
language arts, and include students who are 


assess with alternate achievement standards. 


e Goals for extended graduation rate, if 
defined, must be more rigorous than those 
set for the four year adjusted rate. 


State “levels of performance” for each core 
indicator must be the same for all CTE 
concentrators in the state and, ata minimum 

e be expressed in a percentage or numerical 
form; 

e require the state to continually make 
meaningful progress toward improving the 
performance of all CTE students and 
subgroups; 

e be subject to public comment; 

e when being adjusted pursuant to clause (ii)® 

- take into account how the levels of 
performance involved compare with 
the State levels of performance 
established for other States 

- be higher than the average actual 
performance of the 2 most recently 
completed program years; 

e take into account the extent to which they 
advance the eligible agency’s goals, as set 


7 In this subsection and subsection 

(d), the term ‘subgroup of students’ means— 

“(A) economically disadvantaged students; 

“(B) students from major racial and ethnic groups; 
“(C) children with disabilities; and 

“(D) English language learners 


* Clause (ii) allows states to, prior to the third program year, revise the performance levels for any of the core 
indicators of performance and submit them to the secretary for approval. 
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forth in the state plan. 


Identification Under ESSA and Perkins V 


ESSA° 


Perkins V (Section 123) 


Based on the system of meaningful identification 
the state must identify schools for (CSI) at least 
every three years: 

- atleast the lowest-performing 5% of all 
schools receiving Title 1 funds; 

- all public high schools in the state failing to 
graduate one third or more of their 
students; and 

- schools that were previously identified for 
TSI and did not improve within the state- 
define timeline. 


Schools identified for CSI must develop an 
improvement plan and may be eligible for “more 
rigorous state-determined action” if they do not 
meet exit criteria in a specified number of years. 


Identify annually for targeted support and 
improvement, those schools having one or more 
consistently underperforming sub-groups (as 
defined by the state). 


Schools receiving TSI must develop an 
improvement plan and may be eligible for 
additional action if the LEA’s exit criteria are not 
met. 


Identify at least every three years those schools 
having a subgroup that would, on its own, lead 
to identification for CSI for additional targeted 
support and improvement. 


If a state fails to meet at least 90% of the State 
determined level of performance for any of the 
core indicators of performance, the eligible 
agency shall develop and implement a program 
improvement plan during the first program year 
succeeding the program year for which the 
eligible agency failed to so meet the State 
adjusted determined level of performance for 
any of the core indicators. 


The secretary may withhold funds if the state 
does not implement an improvement plan or 
fails to meet at least 90% of the State 
determined level of performance for any core 
indicator for 2 consecutive years after 
identification 


If an LEA determines that 90% of an agreed 
upon level of proficiency has not been met for 
one or more core indicators by an eligible 
recipient, that recipient shall develop and 
implement an improvement plan 


The LEA may withhold funds if the eligible 
recipient does not implement an improvement 
plan or fails to meet at least 90% of the agreed 
upon level of performance for any core indicator 
for 2 consecutive years after identification 


* Fora comprehensive summary and interpretation of ESSA’s requirements see Lyons, S., D’Brot, J. and Landl, E 
(2018) State Systems of Identification and Support under ESSA: A focus on designing and revising systems of school 


identification at https://ccsso.org/sites/default/files/2017- 


12/State%20Systems%200f%201ID%20and%20Support%20-%20Designing%20and%20Revising%20Systems_0.pdf 
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